On the performance of novice evaluators in usability evaluations

The paper investigates the performance of novice evaluators in usability evaluations by presenting the results of a comparative usability evaluation that was conducted by nine novice evaluator teams. The evaluation teams performed considerably well in terms of the validity of their results, which counts for their participation in usability evaluation projects. The thoroughness of the results obtained was found to be in a relatively stable ratio of about 20-25% of the total number of problems found for eight of the nine teams, which gives a clear indication of the degree to which novice evaluators can identify usability problems. The consistency of the results was not satisfactory, although similar to results of other studies that involve professional evaluators. The paper suggests that when novice evaluators have to be employed for usability evaluations and it is important to find most usability problems then parallel usability evaluations can provide overall valid and thorough results.